{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "be34d25b",
      "metadata": {
        "id": "8377c056591f"
      },
      "source": [
        "Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "6130c8e6",
      "metadata": {
        "cellView": "form",
        "id": "ca23c3f523a7"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "04367b4433c6"
      },
      "source": [
        "# Fine-tune PaliGemma 2 with Keras\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "<td>\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/PaliGemma/[PaliGemma_2]Finetune_with_Keras.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "</td>\n",
        "<td>\n",
        "<a target=\"_blank\" href=\"https://github.com/google-gemini/gemma-cookbook/blob/main/PaliGemma/[PaliGemma_2]Finetune_with_Keras.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "</td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "edb02808",
      "metadata": {
        "id": "afe2be3c68c3"
      },
      "source": [
        "This notebook shows how to fine-tune [PaliGemma 2](https://ai.google.dev/gemma/docs/paligemma) on a vision-language task with [Keras](https://keras.io/). Fine-tuning is a process that can improve your model's performance on specific tasks or help the model adhere to specific output requirements when instructions aren't sufficient and you have a set of examples that demonstrate the outputs you want. Gemma-based models like PaliGemma require fine-tuning to produce expected results.\n",
        "\n",
        "## Before you begin\n",
        "\n",
        "Before going through this notebook, you should be familiar with Python code, as well as how large language models (LLMs) are trained. You don't need to be familiar with Keras, but basic knowledge about Keras (or similar technologies) is helpful when reading through the example code.e."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c941724c",
      "metadata": {
        "id": "dfbbe167352c"
      },
      "source": [
        "## Install KerasHub"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "468e967f",
      "metadata": {
        "id": "5341a3c6ebe7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting einops\r\n",
            "  Downloading einops-0.8.0-py3-none-any.whl.metadata (12 kB)\r\n",
            "Downloading einops-0.8.0-py3-none-any.whl (43 kB)\r\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.2/43.2 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
            "\u001b[?25hInstalling collected packages: einops\r\n",
            "Successfully installed einops-0.8.0\r\n"
          ]
        }
      ],
      "source": [
        "!pip install -q -U keras-nlp\n",
        "!pip install -q -U keras>=3\n",
        "!pip install einops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "b4eadf69",
      "metadata": {
        "id": "973190af9fb4"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U keras keras-hub\n",
        "\n",
        "import os\n",
        "# Set the backbend before importing Keras\n",
        "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
        "# Avoid memory fragmentation on JAX backend.\n",
        "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n",
        "\n",
        "# Set the fine-tuning variables\n",
        "BATCH_SIZE = 1\n",
        "TRAIN_EXAMPLES = 4\n",
        "LEARNING_RATE = 0.003\n",
        "\n",
        "TRAIN_STEPS = TRAIN_EXAMPLES // BATCH_SIZE\n",
        "EVAL_STEPS = TRAIN_STEPS // 4"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b7e49a04",
      "metadata": {
        "id": "0f0cc14c99cb"
      },
      "source": [
        "## Download the training and validation data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "6892fd5b",
      "metadata": {
        "id": "4fde23bb6f99"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_23/4203520384.py:15: DeprecationWarning: Importing display from IPython.core.display is deprecated since IPython 7.14, please import from IPython display\n",
            "  from IPython.core.display import display, HTML\n"
          ]
        }
      ],
      "source": [
        "import io\n",
        "import json\n",
        "import os\n",
        "import urllib\n",
        "\n",
        "import base64\n",
        "import html\n",
        "\n",
        "import numpy as np\n",
        "import keras\n",
        "import keras_hub\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from IPython.core.display import display, HTML\n",
        "from PIL import Image\n",
        "\n",
        "train_file = urllib.request.urlopen(\n",
        "    \"https://storage.googleapis.com/longcap100/data_train90.jsonl\"\n",
        ")\n",
        "val_file = urllib.request.urlopen(\n",
        "    \"https://storage.googleapis.com/longcap100/data_val10.jsonl\"\n",
        ")\n",
        "\n",
        "# Crop the image to the desired dimensions.\n",
        "target_size = (224, 224)\n",
        "\n",
        "def load_image(image_url):\n",
        "    image = tf.io.decode_jpeg(urllib.request.urlopen(image_url).read())\n",
        "    return tf.image.resize(image, size=target_size)\n",
        "\n",
        "def load_dataset(file):\n",
        "    captions = []\n",
        "    images = []\n",
        "    for line in file:\n",
        "        sample = json.loads(line)\n",
        "        captions.append(sample[\"suffix\"])\n",
        "        image_name = sample[\"image\"]\n",
        "        image_url = f\"https://storage.googleapis.com/longcap100/{image_name}\"\n",
        "        images.append(load_image(image_url))\n",
        "    return tf.data.Dataset.from_tensor_slices({\n",
        "        \"images\": images,\n",
        "        \"prompts\": [\"caption en\\n\"] * len(images),\n",
        "        \"responses\": captions,\n",
        "    })\n",
        "\n",
        "train_data = load_dataset(train_file).shuffle(1000).batch(BATCH_SIZE)\n",
        "val_data = load_dataset(val_file).shuffle(1000).batch(BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ccba239c",
      "metadata": {
        "id": "f2480f920144"
      },
      "source": [
        "## View training examples\n",
        "\n",
        "In this notebook, the training data contains 90 images that are paired with long descriptions of what's depicted in the image.\n",
        "\n",
        "**Note:** Normal training data sets that are meant to be used for practical use cases should contain more images, but this notebook limits the number of data points so that you can train the model in a reasonable amount of time for an example.\n",
        "\n",
        "The code below prints a random selection of images with their descriptions from the training data set so that you can see what the images and descriptions your model is trained on looks like. Each image is displayed in as a 128x128 pixel JPEG, with the description printed next to the image to the right."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "cd8d59b7",
      "metadata": {
        "id": "ee549257aa4f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training examples\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A pile of tools sits on a brown carpet, including a green and black cordless drill, a yellow and black electronic device.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A white towel with a Happy Easter message written on it. The towel is white, and the eggs are scattered on the towel. There are four colors of eggs on the towel, yellow, pink, white and blue. The message is written in black, and the letters are black. The towel is clean.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A yellow train is stopped at a station. The train is on the tracks, and the lights are on. The platform has a white line and a white stripe on the platform. There is a person standing on the platform, and a person walking on the platform. The train has a number on its side. The train is moving, and the doors are closed.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A group of people stand next to a train on a platform. The train is parked on the tracks, and the platform is made of bricks. There are several people standing on the platform, including a woman wearing a red shirt, a woman wearing a black shirt, and a woman wearing a blue shirt. The train has a window on the side. The sky is clear, and the sun is shining.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A flamingo stands gracefully in the calm water, its long legs extending gracefully. The flamingo&#x27;s pink feathers and black tail feathers contrast beautifully against the blue water. Its long neck and slender legs are prominent features, while its beak, black as night, and eye, white as the moon, add a touch of whimsy. The flamingo&#x27;s reflection dances on the water&#x27;s surface, mirroring its graceful movement. The flamingo&#x27;s legs are pink.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A surfer rides a wave, his white shirt billowing in the wind. The wave is white and blue, and the water is clear and blue. The surfer&#x27;s board is white and blue, and his hair is dark. He is crouched on the board, his arm extended out to the side. The wave is crashing behind him, and the water is splashing in the air. The surfer is wearing a white shirt and black shorts, and his hair is wet.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A storefront with a sign that reads &quot;Segovia Coffee Bar&quot; sits on a city street. The store has a black awning with white lettering, a white sign on the building, and a window on the building. There is a tree in front of the building,. The sidewalk in front of the store is empty, and there is a shadow on the ground. The store has a window on the building, and the window is open.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A large sign in the shape of a crown stands proudly in the center of a city square. The sign is illuminated by the reflection of the sun on the water, creating a vibrant display. A tall building casts long shadows on the ground, while a flag on top of a building waves proudly. People stroll along the sidewalk. The sky is clear and blue, with fluffy white clouds drifting above. The reflection of the city in the water is a mirror image of the city itself, showcasing the beauty and diversity of this urban landscape.</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def render_inline(image, resize=(224, 224)):\n",
        "  \"\"\"Convert image into inline html.\"\"\"\n",
        "  image = tf.keras.preprocessing.image.array_to_img(image)\n",
        "  image.resize(resize)\n",
        "  with io.BytesIO() as buffer:\n",
        "    image.save(buffer, format='jpeg')\n",
        "    image_b64 = str(base64.b64encode(buffer.getvalue()), \"utf-8\")\n",
        "    return f\"data:image/jpeg;base64,{image_b64}\"\n",
        "\n",
        "def render_example(image, caption):\n",
        "  image = np.asarray(image)\n",
        "  return f\"\"\"\n",
        "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
        "        <img style=\"width:128px; height:128px;\" src=\"{render_inline(image, resize=(64,64))}\" />\n",
        "        <p style=\"width:256px; margin:10px; font-size:small;\">{html.escape(caption)}</p>\n",
        "    </div>\n",
        "    \"\"\"\n",
        "\n",
        "html_out = \"\"\n",
        "\n",
        "for element in train_data.take(8):\n",
        "  caption = tf.compat.as_str_any(element[\"responses\"].numpy()[0])\n",
        "  html_out += render_example(element[\"images\"].numpy()[0], caption)\n",
        "\n",
        "print(\"Training examples\")\n",
        "display(HTML(html_out))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "53c0034b",
      "metadata": {
        "id": "e90446ab2251"
      },
      "source": [
        "## Load Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "10967820",
      "metadata": {
        "id": "39c82b20d38a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"pali_gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"pali_gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaTokenizer</span>)                     │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">257,152</span> │\n",
              "├───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤\n",
              "│ pali_gemma_image_converter (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaImageConverter</span>)          │                   Image size: (<span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>) │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (\u001b[38;5;33mPaliGemmaTokenizer\u001b[0m)                     │                      Vocab size: \u001b[38;5;34m257,152\u001b[0m │\n",
              "├───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤\n",
              "│ pali_gemma_image_converter (\u001b[38;5;33mPaliGemmaImageConverter\u001b[0m)          │                   Image size: (\u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m) │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"pali_gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"pali_gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>)       │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">3,032,094,960</span> │ images[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],              │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaBackbone</span>)           │                           │                 │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│                               │                           │                 │ response_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
              "│                               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">592,478,208</span> │ pali_gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]  │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GetItem</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ token_embedding[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (\u001b[38;5;33mInputLayer\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m3\u001b[0m)       │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (\u001b[38;5;33mInputLayer\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m3,032,094,960\u001b[0m │ images[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],              │\n",
              "│ (\u001b[38;5;33mPaliGemmaBackbone\u001b[0m)           │                           │                 │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│                               │                           │                 │ response_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
              "│                               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │     \u001b[38;5;34m592,478,208\u001b[0m │ pali_gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]  │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (\u001b[38;5;33mGetItem\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │               \u001b[38;5;34m0\u001b[0m │ token_embedding[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,032,094,960</span> (11.30 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m3,032,094,960\u001b[0m (11.30 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,032,094,960</span> (11.30 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m3,032,094,960\u001b[0m (11.30 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "pali_gemma_lm = keras_hub.models.PaliGemmaCausalLM.from_preset(\n",
        "    \"/kaggle/input/paligemma2/keras/pali_gemma2_pt_3b_224/1\"\n",
        "#    \"kaggle://keras/paligemma2/keras/pali_gemma2_pt_3b_224\"\n",
        ")\n",
        "pali_gemma_lm.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dd4fe0c2",
      "metadata": {
        "id": "edeab21cb990"
      },
      "source": [
        "## Inference before fine tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "6355a8e9",
      "metadata": {
        "id": "ab780f60d802"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "a cow on the beach</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Inference Result\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "a series of white hangers</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "person in a red blazer and black pants</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "the top is a mix of fabrics .</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "love will save us sweatshirt in pink</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "test_image_url = 'https://storage.googleapis.com/keras-cv/models/paligemma/cow_beach_1.png'\n",
        "test_image = load_image(test_image_url)\n",
        "\n",
        "def inference_test(image):\n",
        "  prompt = 'caption en\\n'\n",
        "  output = pali_gemma_lm.generate(\n",
        "      inputs={\n",
        "          \"images\": image,\n",
        "          \"prompts\": prompt,\n",
        "      }\n",
        "  )\n",
        "  return render_example(image, output)\n",
        "\n",
        "display(HTML(inference_test(test_image)))\n",
        "\n",
        "\n",
        "def make_predictions():\n",
        "  html_out = \"\"\n",
        "  for element in val_data.take(4):\n",
        "    html_out += inference_test(element[\"images\"].numpy()[0])\n",
        "\n",
        "  print(\"\\nInference Result\")\n",
        "  display(HTML(html_out))\n",
        "\n",
        "make_predictions()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eaea17d7",
      "metadata": {
        "id": "7ba37a64794f"
      },
      "source": [
        "## LoRA Fine-tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "40d24252",
      "metadata": {
        "id": "0a61883f4a81"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"pali_gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"pali_gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaTokenizer</span>)                     │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">257,152</span> │\n",
              "├───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤\n",
              "│ pali_gemma_image_converter (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaImageConverter</span>)          │                   Image size: (<span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>) │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (\u001b[38;5;33mPaliGemmaTokenizer\u001b[0m)                     │                      Vocab size: \u001b[38;5;34m257,152\u001b[0m │\n",
              "├───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤\n",
              "│ pali_gemma_image_converter (\u001b[38;5;33mPaliGemmaImageConverter\u001b[0m)          │                   Image size: (\u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m) │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"pali_gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"pali_gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>)       │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">3,035,023,600</span> │ images[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],              │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaBackbone</span>)           │                           │                 │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│                               │                           │                 │ response_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
              "│                               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">592,478,208</span> │ pali_gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]  │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GetItem</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ token_embedding[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (\u001b[38;5;33mInputLayer\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m3\u001b[0m)       │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (\u001b[38;5;33mInputLayer\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m3,035,023,600\u001b[0m │ images[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],              │\n",
              "│ (\u001b[38;5;33mPaliGemmaBackbone\u001b[0m)           │                           │                 │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│                               │                           │                 │ response_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
              "│                               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │     \u001b[38;5;34m592,478,208\u001b[0m │ pali_gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]  │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (\u001b[38;5;33mGetItem\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │               \u001b[38;5;34m0\u001b[0m │ token_embedding[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,035,023,600</span> (11.31 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m3,035,023,600\u001b[0m (11.31 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,928,640</span> (11.17 MB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">3,032,094,960</span> (11.30 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m3,032,094,960\u001b[0m (11.30 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Enable lora to freeze most of the model and save memory.\n",
        "pali_gemma_lm.backbone.enable_lora(4)\n",
        "pali_gemma_lm.summary()\n",
        "\n",
        "# Lower our sequence length to further save memory.\n",
        "pali_gemma_lm.preprocessor.sequence_length = 64\n",
        "\n",
        "# Use Cosine Decay Scheduler with Warm up\n",
        "#cosine_decay_scheduler = tf.keras.optimizers.schedules.CosineDecay(\n",
        "#    initial_learning_rate = 0,\n",
        "#    decay_steps = TRAIN_EXAMPLES,\n",
        "#    warmup_target = LEARNING_RATE,\n",
        "#    warmup_steps = TRAIN_EXAMPLES / 10\n",
        "#)\n",
        "\n",
        "def plot_scheduler(step, scheduler):\n",
        "  x = range(step)\n",
        "  y = []\n",
        "  for step in x:\n",
        "    y.append(scheduler(step))\n",
        "  plt.plot(x, y, label=scheduler.name)\n",
        "  plt.xlabel('Epoch')\n",
        "  plt.ylabel('Learning Rate')\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "\n",
        "#plot_scheduler(TRAIN_EXAMPLES, cosine_decay_scheduler)\n",
        "\n",
        "# Use AdamW (a common optimizer for transformer models).\n",
        "#optimizer = keras.optimizers.SGD(learning_rate=cosine_decay_scheduler)\n",
        "optimizer = keras.optimizers.AdamW(learning_rate=LEARNING_RATE)\n",
        "\n",
        "pali_gemma_lm.compile(\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    optimizer=optimizer,\n",
        "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "da88646d",
      "metadata": {
        "id": "016ed3975263"
      },
      "source": [
        "## Fine-tune the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "56c00057",
      "metadata": {
        "id": "17867028751a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m117s\u001b[0m 877ms/step - loss: 1.6715 - sparse_categorical_accuracy: 0.5523\n",
            "Epoch 2/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m80s\u001b[0m 567ms/step - loss: 0.9483 - sparse_categorical_accuracy: 0.6871\n",
            "Epoch 3/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m52s\u001b[0m 555ms/step - loss: 0.7245 - sparse_categorical_accuracy: 0.7587\n",
            "Epoch 4/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m52s\u001b[0m 555ms/step - loss: 0.5710 - sparse_categorical_accuracy: 0.8049\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA86ElEQVR4nO3de1xUdeL/8fcMdxTwjiB4zxsigqVptWpZpmRZbVnaan2335apZda22ppmtVFbW7aF3bZyK69dtFLzEoZmWa4C3kC8gOIF8A6Ccp3z+4OkKNEZFM4M83o+HucPh8/hvDnNY+bdnM+cj8UwDEMAAAAmsZodAAAAuDfKCAAAMBVlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqSgjAADAVJ5mB7CHzWbToUOHFBAQIIvFYnYcAABgB8MwdOrUKYWGhspqrf7zD5coI4cOHVJ4eLjZMQAAQA3s379fYWFh1f7cJcpIQECApIo/JjAw0OQ0AADAHvn5+QoPD698H6+OS5SRs5dmAgMDKSMAALiYC02xYAIrAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKZy6zKy/VCe7n7nRx0vLDE7CgAAbstty4jNZuixhZu1PuOYJi1Mkc1mmB0JAAC35LZlxGq16NURPeXjaVVi+hHNStxtdiQAANyS25YRSeoaEqhnh3eXJL2yaqd+2HPU5EQAALgfty4jknTn5eG6o1eYbIb08LwUHc4vMjsSAABuxe3LiCQ9c0t3dWkZoKMFxRo/L1ll5TazIwEA4DYoI5L8vD00a1SMGvp4akPmcb28cqfZkQAAcBuUkZ+1b95QL97eQ5L01po9SkjLNTkRAADugTLyK7E9QnRvv7aSpEkLN2v/8dPmBgIAwA1QRn7jyaFd1TO8kfLOlGrc3CQVl5WbHQkAgHqNMvIb3p5WxY+KUSN/L205kKd/LE0zOxIAAPUaZeQcWjXy06sjekqSPly/T19tPmRuIAAA6jHKSDUGdm6hcQM7SJImf7ZFe44UmJwIAID6iTJyHo8O6qQr2zdRYUm5xn68SadLysyOBABAvUMZOQ9PD6v+fXe0mgf4aGdugaYu3ibDYEE9AAAuJcrIBbQI8NXrd0fLapE+TzqoBf/bb3YkAADqFcqIHa5s31SPD+4sSZr25XZtP5RnciIAAOoPyoidHvxDB13XpYVKymx6aE6S8otKzY4EAEC9QBmxk9Vq0b/ujFKrRn7ad+y0nvhkC/NHAAC4BCgjDmjk7634UTHy8rBo+fYcvf/9XrMjAQDg8igjDuoZ3khTY7tJkuKWpWnTvhMmJwIAwLVRRmpgdN82iu0RojKbofFzk3S8sMTsSAAAuCyHy8jatWs1bNgwhYaGymKxaPHixXbv+/3338vT01M9e/Z09LBOxWKx6MXbe6h98wbKzivSxAUpstmYPwIAQE04XEYKCwsVFRWl+Ph4h/Y7efKkRo8ereuuu87RQzqlhj6eenNUL/l6WbV25xG98e1usyMBAOCSHC4jQ4YM0XPPPadbb73Vof0efPBBjRw5Un379nX0kE6rc8sAPTc8UpL06jc79f3uoyYnAgDA9dTJnJEPPvhAGRkZmj59el0crk79sVeYRlweLsOQHpmfrNz8IrMjAQDgUmq9jOzatUuTJ0/Wxx9/LE9PT7v2KS4uVn5+fpXNmc24JUJdQwJ1tKBEE+Ymq6zcZnYkAABcRq2WkfLyco0cOVIzZsxQp06d7N4vLi5OQUFBlVt4eHgtprx4vl4emjUqRg19PLVh73G9tDLd7EgAALgMi3ERtxG1WCxatGiRhg8ffs6fnzx5Uo0bN5aHh0flYzabTYZhyMPDQytXrtS11177u/2Ki4tVXFxc+e/8/HyFh4crLy9PgYGBNY1b677emq2xc5IkSe+OvlzXdws2OREAAObJz89XUFDQBd+/7btuUkOBgYHaunVrlcdmzZql1atX69NPP1W7du3OuZ+Pj498fHxqM1qtGBIZovuuaqsPvt+rxxamaOnD1yi8ib/ZsQAAcGoOl5GCggLt3v3L11gzMzOVkpKiJk2aqHXr1poyZYoOHjyoDz/8UFarVd27d6+yf4sWLeTr6/u7x+uLKUO6KmX/SSVnndRDc5L0yYN95evlceEdAQBwUw7PGdm4caOio6MVHR0tSZo0aZKio6M1bdo0SVJ2draysrIubUoX4u1pVfzIGDX299LWg3l6bmmq2ZEAAHBqFzVnpK7Ye83JmSSmH9Z9s/8nw5Beu6unbunZyuxIAADUKXvfv1mbppYM6NxC4wd2lCRN+Xyrdh8+ZXIiAACcE2WkFk0c1En9OjTV6ZJyjf04SadLysyOBACA06GM1CIPq0Wv3RWtFgE+2nW4QH9ftE0ucFUMAIA6RRmpZc0DfPT63dHysFq0KPmg5m3Yb3YkAACcCmWkDvRp31SP39BZkvT0V9u17WCeyYkAAHAelJE68sAf2mtQ1xYqKbPpoTlJyjtTanYkAACcAmWkjlitFv3rjp4Ka+ynrOOn9ddPNjN/BAAAUUbqVJC/l2aNipG3h1UrU3P13rpMsyMBAGA6ykgd6xHWSE/d1FWS9MLXO7Rp33GTEwEAYC7KiAnuubKNhkWFqsxmaNycZB0rKL7wTgAA1FOUERNYLBbF3Rap9s0bKCe/SBMXpKjcxvwRAIB7ooyYpKGPp94c1Uu+XlZ9t+uoXl+9y+xIAACYgjJios4tA/T8rZGSpNcSdum7XUdMTgQAQN2jjJjstpgw3d07XIYhTZyfopy8IrMjAQBQpygjTmD6sAh1CwnUscISjZ+bpNJym9mRAACoM5QRJ+Dr5aE374lRgI+nNu47oZdWpJsdCQCAOkMZcRJtmjbQS3f0kCS9szZDK7fnmJwIAIC6QRlxIjd2D9Gfr24nSXrsk83KOnba5EQAANQ+yoiTmTyki2JaN9KpojKNnbNJRaXlZkcCAKBWUUacjJeHVW+MjFFjfy9tP5SvZ5akmh0JAIBaRRlxQqGN/DTzrmhZLNLcn7K0OPmg2ZEAAKg1lBEn1b9Tc0249jJJ0pTPt2pX7imTEwEAUDsoI07skesu09Udm+lMabnGzklSYXGZ2ZEAALjkKCNOzMNq0cy7eio40Ee7Dxfo74u2yjBYUA8AUL9QRpxcs4Y+emNkjDysFi1OOaS5G7LMjgQAwCVFGXEBV7RtoicGd5YkzfgyVVsP5JmcCACAS4cy4iL+8of2GtQ1WCXlNj00d5PyTpeaHQkAgEuCMuIiLBaL/nVHlMIa+2n/8TN6/NPNzB8BANQLlBEXEuTvpTdH9ZK3h1WrUnP17ncZZkcCAOCiUUZcTGRYkKYN6yZJenF5uv6397jJiQAAuDiUERc0qk9r3dIzVOU2Q+PnJuloQbHZkQAAqDGHy8jatWs1bNgwhYaGymKxaPHixecdv27dOl111VVq2rSp/Pz81KVLF7366qs1zQtVzB95/tZIdWzRULn5xZo4P0XlNuaPAABck8NlpLCwUFFRUYqPj7drfIMGDTR+/HitXbtWaWlpmjp1qqZOnap33nnH4bD4RQMfT705KkZ+Xh5at/uoXkvYZXYkAABqxGJcxFcyLBaLFi1apOHDhzu032233aYGDRroo48+smt8fn6+goKClJeXp8DAwBokrb8WJR/Qows2y2KR/ntfb/2hU3OzIwEAIMn+9+86nzOSnJysH374Qf3796/rQ9dLt0aH6e7erWUY0sQFKcrOO2N2JAAAHFJnZSQsLEw+Pj66/PLLNW7cON1///3Vji0uLlZ+fn6VDdWbPqybIkIDdbywROPnJqu03GZ2JAAA7FZnZeS7777Txo0b9dZbb2nmzJmaN29etWPj4uIUFBRUuYWHh9dVTJfk6+WhN0f1UoCvpzbtO6EXv95hdiQAAOxmypyR5557Th999JHS09PP+fPi4mIVF//yddX8/HyFh4czZ+QCVmzP0QMfbZIkvXVPL93YvaXJiQAA7sxp54xIks1mq1I2fsvHx0eBgYFVNlzY4IiW+n/XtJMk/fWTzdp3rNDkRAAAXJinozsUFBRo9+7dlf/OzMxUSkqKmjRpotatW2vKlCk6ePCgPvzwQ0lSfHy8WrdurS5dukiquE/Jyy+/rIcffvgS/Qn4tSdu7KLkrJPauO+Exn6cpM8f6idfLw+zYwEAUC2Hy8jGjRs1cODAyn9PmjRJkjRmzBjNnj1b2dnZysrKqvy5zWbTlClTlJmZKU9PT3Xo0EEvvviiHnjggUsQH7/l5WHV6yOjFfvvdUrNzteMr7Yr7rYeZscCAKBaFzVnpK5wnxHHfbfriEa/v0GGIb1yZ5RuiwkzOxIAwM049ZwR1L5rLmuuh6+9TJL090XbtDP3lMmJAAA4N8pIPfbwdZfpmsua6UxpucZ+vEmFxWVmRwIA4HcoI/WYh9WimSN6qmWgr/YcKdSUz7fKBa7KAQDcDGWknmva0EdvjIyWh9WiLzcf0sc/ZV14JwAA6hBlxA1c3raJJt9Y8dXqZ79K1ZYDJ80NBADAr1BG3MT917TTDd2CVVJu00NzkpR3utTsSAAASKKMuA2LxaKX7ohS6yb+OnDijB77JEU2G/NHAADmo4y4kSA/L80aFSNvT6u+STusd77LMDsSAACUEXfTvVWQpg/rJkl6aUW6fso4ZnIiAIC7o4y4oZG9W+vW6FYqtxmaMC9ZR05Vv2ghAAC1jTLihiwWi/5xa3dd1qKhDp8q1iPzk1XO/BEAgEkoI27K39tTb94TI39vD/2w55he+2an2ZEAAG6KMuLGOrYIUNxtkZKkf6/ercT0wyYnAgC4I8qIm7ulZyuN6tNakvToghQdOnnG5EQAAHdDGYGeuqmburcK1InTpRo3N0klZTazIwEA3AhlBPL18tCskb0U4Oup5KyTeuHrHWZHAgC4EcoIJEmtm/rrlTt7SpLe/z5TX2/NNjcQAMBtUEZQ6fpuwXrgD+0lSU98ukV7jxaanAgA4A4oI6ji8cGddUXbxjpVXKaxc5JUVFpudiQAQD1HGUEVXh5WvX53jJo28FZadr6e/nK72ZEAAPUcZQS/0zLIV6/dFS2LRZr/v/36dNMBsyMBAOoxygjO6erLmmnidZ0kSVMXb9WOnHyTEwEA6ivKCKo14dqOuuayZioqtemhOUkqKC4zOxIAoB6ijKBaVqtFM0f0VMtAX2UcKdTkz7bIMFhQDwBwaVFGcF5NG/ooflS0PK0WLdmSrY9+3Gd2JABAPUMZwQX1atNEk4d0kSQ9uyRVm/efNDcQAKBeoYzALn++up1ujGip0nJDD81J0snTJWZHAgDUE5QR2MViseifd/RQm6b+OnjyjCYt3CybjfkjAICLRxmB3QJ9vTRrVIy8Pa1aveOw3lq7x+xIAIB6gDICh0SEBmnGzRGSpJdXpOvHjGMmJwIAuDrKCBx21xXhui26lWyGNGFesg6fKjI7EgDAhVFG4DCLxaLnbu2uTsENdeRUsR6Zl6Jy5o8AAGrI4TKydu1aDRs2TKGhobJYLFq8ePF5x3/++ee6/vrr1bx5cwUGBqpv375asWJFTfPCSfh7e2rWqF5q4O2h9RnH9OqqnWZHAgC4KIfLSGFhoaKiohQfH2/X+LVr1+r666/XsmXLtGnTJg0cOFDDhg1TcnKyw2HhXDq2aKi423tIkt74dre+TT9sciIAgCuyGBdxf2+LxaJFixZp+PDhDu0XERGhESNGaNq0aXaNz8/PV1BQkPLy8hQYGFiDpKhNTy3epo9+3KdG/l5a+vA1atXIz+xIAAAnYO/7d53PGbHZbDp16pSaNGlS7Zji4mLl5+dX2eC8pt7UVT3CgnTydKnGzUlSSZnN7EgAABdS52Xk5ZdfVkFBge68885qx8TFxSkoKKhyCw8Pr8OEcJSPp4fiR8Yo0NdTKftP6vllaWZHAgC4kDotI3PnztWMGTO0cOFCtWjRotpxU6ZMUV5eXuW2f//+OkyJmghv4q9X7uwpSZr9w14t3ZJtbiAAgMuoszIyf/583X///Vq4cKEGDRp03rE+Pj4KDAysssH5DeoWrAf6t5ck/e2zLco4UmByIgCAK6iTMjJv3jzdd999mjdvnmJjY+vikDDJX2/orN7tmqiguEwPzUlSUWm52ZEAAE7O4TJSUFCglJQUpaSkSJIyMzOVkpKirKwsSRWXWEaPHl05fu7cuRo9erT+9a9/qU+fPsrJyVFOTo7y8vIuzV8Ap+LpYdUbd0erWUNv7cg5pWlfbDM7EgDAyTlcRjZu3Kjo6GhFR0dLkiZNmqTo6OjKr+lmZ2dXFhNJeuedd1RWVqZx48YpJCSkcnvkkUcu0Z8AZ9Mi0Ff/vitaVou0cOMBLdzInB8AQPUu6j4jdYX7jLim1xN26V+rdsrH06rF465S1xD+2wGAO3Ha+4zAfYwb2FH9OzVXcZlND81J0qmiUrMjAQCcEGUEtcZqtejVET0VEuSrzKOFmvzZVrnAB3EAgDpGGUGtatLAW2+MjJGn1aKlW7P13x/2mh0JAOBkKCOodb3aNNaUoV0lSf9YlqbkrBMmJwIAOBPKCOrE/13VVkO6t1RpuaHxc5N1orDE7EgAACdBGUGdsFgsevGPPdS2qb8OnjyjSQtTZLMxfwQAQBlBHQr09dKsUb3k42nVt+lH9OaaPWZHAgA4AcoI6lS30EA9c0uEJOlfK9P1w56jJicCAJiNMoI6d+fl4bo9Jkw2Q3p4XooO5xeZHQkAYCLKCOqcxWLRc8O7q3NwgI4WFGvCvGSVldvMjgUAMAllBKbw8/bQrHti1MDbQz9lHtcrq3aaHQkAYBLKCEzToXlDvXB7D0nSrMQ9Wr0j1+REAAAzUEZgqmFRoRrTt40k6dEFm3XgxGmTEwEA6hplBKZ7MrarosKClHemVOPmJKm4rNzsSACAOkQZgel8PD0UPypGQX5e2nwgT88vTTM7EgCgDlFG4BTCGvvr1RFRkqT/rt+nrzYfMjkRAKCuUEbgNK7tEqyxAzpIkiZ/tkV7jhSYnAgAUBcoI3Aqj13fSX3aNVFhSbke+jhJZ0qYPwIA9R1lBE7F08Oq1++OVrOGPkrPPaWnvthmdiQAQC2jjMDptAj01b/v7imrRfp00wEt/N9+syMBAGoRZQROqV+HZnrshs6SpKe+2KbUQ/kmJwIA1BbKCJzW2P4dNLBzcxWX2fTQnE3KLyo1OxIAoBZQRuC0rFaLXrmzp1o18tPeY6f1t0+3yDAMs2MBAC4xygicWuMG3npjZLS8PCz6eluOPvh+r9mRAACXGGUETi+6dWM9ObSrJOn5ZWlKyjphciIAwKVEGYFLuLdfW8VGhqjMZmj8nCSdKCwxOxIA4BKhjMAlWCwWvXB7pNo1a6BDeUV6dGGKbDbmjwBAfUAZgcsI8PXSrFEx8vG0KjH9iGYl7jY7EgDgEqCMwKV0DQnUs8O7S5JeWbVTP+w+anIiAMDFoozA5dx5ebju6BUmmyE9PD9ZuflFZkcCAFwEyghc0jO3dFeXlgE6WlCiCfOSVVZuMzsSAKCGHC4ja9eu1bBhwxQaGiqLxaLFixefd3x2drZGjhypTp06yWq1auLEiTWMCvzCz9tDs0bFqKGPpzZkHtfLK3eaHQkAUEMOl5HCwkJFRUUpPj7ervHFxcVq3ry5pk6dqqioKIcDAtVp37yhXry9hyTprTV7lJCWa3IiAEBNeDq6w5AhQzRkyBC7x7dt21avvfaaJOn999939HDAecX2CNH/9rbV7B/2atLCzVoy4WqFN/E3OxYAwAFOOWekuLhY+fn5VTagOk8O7aqe4Y2Ud6ZU4+Ymqbis3OxIAAAHOGUZiYuLU1BQUOUWHh5udiQ4MW9Pq+JHxaiRv5e2HMjTc0vSzI4EAHCAU5aRKVOmKC8vr3Lbv3+/2ZHg5Fo18tOrI3pKkj76cZ++3HzI3EAAALs5ZRnx8fFRYGBglQ24kIGdW2jcwA6SpMmfbdHuwwUmJwIA2MMpywhQU48O6qQr2zfR6ZJyPTRnk06XlJkdCQBwAQ6XkYKCAqWkpCglJUWSlJmZqZSUFGVlZUmquMQyevToKvucHV9QUKAjR44oJSVFqampF58e+A1PD6v+fXe0mgf4aGdugaYu3ibDYEE9AHBmFsPBV+rExEQNHDjwd4+PGTNGs2fP1r333qu9e/cqMTHxl4NYLL8b36ZNG+3du9euY+bn5ysoKEh5eXlcsoFd1u85plH/+VE2Q3rhtkjd1bu12ZEAwO3Y+/7tcBkxA2UENRH/7W69tCJd3p5WLXqonyJCg8yOBABuxd73b+aMoN4a27+Dru3SQiVlNj00J0n5RaVmRwIAnANlBPWW1WrRK3dGqVUjP+07dlp//WQz80cAwAlRRlCvNfL3VvyoGHl5WLRie67eW5dpdiQAwG9QRlDv9QxvpKmx3SRJL3y9Q5v2HTc5EQDg1ygjcAuj+7ZRbI8QldkMjZ+brOOFJWZHAgD8jDICt2CxWPTi7T3UvlkDZecVaeKCFNlszB8BAGdAGYHbaOjjqVn3xMjXy6q1O4/ojW93mx0JACDKCNxMl5aBem54pCTp1W92at2uoyYnAgBQRuB2/tgrTCMuD5dhSI/MT1ZOXpHZkQDArVFG4JZm3BKhriGBOlZYognzklRabjM7EgC4LcoI3JKvl4dmjYpRQx9P/W/vCb28It3sSADgtigjcFvtmjXQS3/sIUl6e22GVqXmmpwIANwTZQRubUhkiO67qq0k6bGFKdp//LS5gQDADVFG4PamDOmq6NaNlF9UpofmJKmotNzsSADgVigjcHvenla9MTJGjfy9tPVgnp5bmmp2JABwK5QRQFKrRn6aOaKnLBbp4x+z9EXKQbMjAYDboIwAPxvQuYXGD+woSZry+VbtPnzK5EQA4B4oI8CvTBzUSf06NNXpknKN/ThJp0vKzI4EAPUeZQT4FQ+rRa/dFa0WAT7adbhAf1+0TYbBgnoAUJsoI8BvNA/w0et3R8vDatGi5IOat2G/2ZEAoF6jjADn0Kd9Uz1+Q2dJ0tNfbde2g3kmJwKA+osyAlTjgT+013VdWqikzKaH5iQp70yp2ZEAoF6ijADVsFot+tedUWrVyE9Zx0/rr59sZv4IANQCyghwHo38vfXmPTHy9rBqZWqu/vNdptmRAKDeoYwAF9AjrJGeuqmrJOmF5Tu0ce9xkxMBQP1CGQHscM+VbTQsKlTlNkPj5ybrWEGx2ZEAoN6gjAB2sFgsirstUu2bN1BOfpEmLkhRuY35IwBwKVBGADs19PHUm6N6ydfLqu92HdXrq3eZHQkA6gXKCOCAzi0D9I/hkZKk1xJ26btdR0xOBACujzICOOj2XmG664pwGYb0yPwUZeedMTsSALg0yghQA0/fHKFuIYE6Xlii8XOTVVpuMzsSALgsh8vI2rVrNWzYMIWGhspisWjx4sUX3CcxMVExMTHy8fFRx44dNXv27BpEBZyHr5eH3rwnRgE+ntq074T+uXyH2ZEAwGU5XEYKCwsVFRWl+Ph4u8ZnZmYqNjZWAwcOVEpKiiZOnKj7779fK1ascDgs4EzaNG2gl+7oIUl697tMrdieY3IiAHBNFuMi7m9tsVi0aNEiDR8+vNoxf/vb37R06VJt27at8rG77rpLJ0+e1PLly+06Tn5+voKCgpSXl6fAwMCaxgVqxbNLUvXeukwF+Hpq6YRr1Lqpv9mRAMAp2Pv+XetzRtavX69BgwZVeWzw4MFav359tfsUFxcrPz+/ygY4q8lDuiimdSOdKirT2DmbVFRabnYkAHAptV5GcnJyFBwcXOWx4OBg5efn68yZc38LIS4uTkFBQZVbeHh4bccEaszLw6o3Rsaosb+Xth/K1zNLUs2OBAAuxSm/TTNlyhTl5eVVbvv37zc7EnBeoY38NPOuaFks0tyfsrQo+YDZkQDAZdR6GWnZsqVyc3OrPJabm6vAwED5+fmdcx8fHx8FBgZW2QBn179Tc00Y2FGS9OTn27Qjh8uLAGCPWi8jffv2VUJCQpXHVq1apb59+9b2oYE698igTrqqY1OdKS3X8Pjv9XrCLuaQAMAFOFxGCgoKlJKSopSUFEkVX91NSUlRVlaWpIpLLKNHj64c/+CDDyojI0NPPPGEduzYoVmzZmnhwoV69NFHL81fADgRD6tF/74rWr3bNVFRqU3/WrVTN7y6Vt+k5uoivrgGAPWaw1/tTUxM1MCBA3/3+JgxYzR79mzde++92rt3rxITE6vs8+ijjyo1NVVhYWF66qmndO+999p9TL7aC1djGIa+3HxIzy9LU25+sSRpQOfmmnZTN7Vv3tDkdABQN+x9/76o+4zUFcoIXFVhcZleX71b763LUGm5IS8Pi/58dXtNuLajGvh4mh0PAGoVZQRwIhlHCvTMklQlples8tsy0FdThnbRzVEVyyoAQH1EGQGcjGEYSkg7rGeWpCrr+GlJUu92TTTj5gh1DeF5DaD+oYwATqqotFzvrs1QfOJuFZXaZLVI91zZRpOu76RG/t5mxwOAS4YyAji5gyfP6PmlaVq6NVuS1NjfS0/c2EV3Xh4uDyuXbgC4PsoI4CJ+2H1UT3+1XTtzCyRJka2CNOOWCMW0bmxyMgC4OJQRwIWUltv04fp9mrlqp04Vl0mSbo8J09+GdFaLAF+T0wFAzVBGABd05FSx/rl8hz7ZVLG2TYCPpx4ZdJnG9GsrLw+nXEoKAKpFGQFcWHLWCU3/cru2HMiTJHVs0VAzbo7QVR2bmZwMAOxHGQFcnM1maOHG/frninQdLyyRJA3p3lJ/j+2qsMb+JqcDgAujjAD1RN7pUr36zU59uH6vbIbk62XV2P4d9UD/9vL18jA7HgBUizIC1DNp2fma/uV2bcg8LkkKb+Knp2K76fpuwdzFFYBToowA9ZBhGPpqS7aeX5qmnPwiSdIfOjXX9GHd1IEF+AA4GcoIUI8VFpcp/tvd+s93mSopt8nLw6L/u6qdJlx3mRqyAB8AJ0EZAdxA5tFCPbskVat3HJYktQjw0ZNDu+qWnizAB8B8lBHAjSSk5eqZJanad6xiAb4r2jbW0zdHKCI0yORkANwZZQRwM0Wl5XpvXabeWL1bZ0rLZbVII/u01uM3dGYBPgCmoIwAburQyTN6flmalmypWICvkb+XHr+hs+7u3ZoF+ADUKcoI4OZ+2HNUM75MVXruKUlSRGignrklQr3aNDE5GQB3QRkBoLJymz76cZ9eWbVTp4oqFuC7LbqVJg/pohaBLMAHoHZRRgBUOlpQrJeWp2vhpv0yDKmhj6cevq6j7u3XTt6eLMAHoHZQRgD8Tsr+k5r+xTZt/nkBvg7NG+jpmyN0zWXNTU4GoD6ijAA4J5vN0KebDujF5Tt07OcF+AZHBGtqbDeFN2EBPgCXDmUEwHnlnSnVzG926sP1+1RuM+TjadWD/Tto7IAOLMAH4JKgjACwS3rOKU3/cpt+zKhYgK9VIz89dVM3DY5gAT4AF4cyAsBuhmFo6dZs/WNpmrLzKhbgu+ayZpo+LEIdW7AAH4CaoYwAcNjpkjLN+naP3lmboZJymzytFt13VVs9fN1lCvD1MjseABdDGQFQY/uOVSzA901axQJ8zQN8NPnGLro1upWs3MUVgJ0oIwAu2rc7DmvGV9u19+cF+Hq1aawZN0eoeysW4ANwYZQRAJdEcdkvC/CdLimXxSLd3bu1/npDZzVuwAJ8AKpHGQFwSWXnnVHcsh36cvMhSVKQn5ceH9xZI1mAD0A1KCMAasWPGcf09JfbtSOnYgG+biGBmnFLhK5oywJ8AKqy9/27RotSxMfHq23btvL19VWfPn20YcOGaseWlpbqmWeeUYcOHeTr66uoqCgtX768JocF4ASubN9USyZcrRk3RyjQ11Op2fm64631mjg/Wbn5RWbHA+CCHC4jCxYs0KRJkzR9+nQlJSUpKipKgwcP1uHDh885furUqXr77bf1+uuvKzU1VQ8++KBuvfVWJScnX3R4AObw9LBqTL+2+vbxAbq7d7gsFmlxyiFd+3Ki3lqzRyVlNrMjAnAhDl+m6dOnj6644gq98cYbkiSbzabw8HBNmDBBkydP/t340NBQ/f3vf9e4ceMqH7v99tvl5+enjz/+2K5jcpkGcG5bDpzUtC+2K2X/SUlS+2YNNP3mCPXvxAJ8gDurlcs0JSUl2rRpkwYNGvTLL7BaNWjQIK1fv/6c+xQXF8vX17fKY35+flq3bl21xykuLlZ+fn6VDYDz6hHWSJ+P7aeX/thDzRp6K+Nooca8v0H/78ONyvr5a8EAUB2HysjRo0dVXl6u4ODgKo8HBwcrJyfnnPsMHjxYr7zyinbt2iWbzaZVq1bp888/V3Z2drXHiYuLU1BQUOUWHh7uSEwAJrBaLbrj8nCtfnyA/nx1O3lYLVqVmqtBr67RKyvTdaak3OyIAJxUjSawOuK1117TZZddpi5dusjb21vjx4/XfffdJ6u1+kNPmTJFeXl5ldv+/ftrOyaASyTQ10tP3dRNyx+5Rld1bKqSMpv+vXq3Br2yRsu2ZssFvsAHoI45VEaaNWsmDw8P5ebmVnk8NzdXLVu2POc+zZs31+LFi1VYWKh9+/Zpx44datiwodq3b1/tcXx8fBQYGFhlA+BaLgsO0Md/7qNZo2LUqpGfDp48o4fmJOme937SrtxTZscD4EQcKiPe3t7q1auXEhISKh+z2WxKSEhQ3759z7uvr6+vWrVqpbKyMn322We65ZZbapYYgMuwWCwaGhmibyb118PXdpS3p1Xf7z6mIa99p2eXpCq/qNTsiACcgMOXaSZNmqR3331X//3vf5WWlqaxY8eqsLBQ9913nyRp9OjRmjJlSuX4n376SZ9//rkyMjL03Xff6cYbb5TNZtMTTzxx6f4KAE7Nz9tDk27orG8e7a/ruwWrzGbovXWZuvblNfp00wHZbFy6AdyZp6M7jBgxQkeOHNG0adOUk5Ojnj17avny5ZWTWrOysqrMBykqKtLUqVOVkZGhhg0baujQofroo4/UqFGjS/ZHAHANrZv6693Rlysx/bCe+SpVGUcL9fgnmzXnp3165ubuigxjAT7AHXE7eACmKCmz6f3vM/V6wi4V/rwA311XhOuvg7uoCQvwAfUCa9MAcAm5+UWKW5amxSkVC/AF+npWLsDn6VHrX/gDUIsoIwBcyobM45r+5XalZVfc5LBLywDNuDlCfdo3NTkZgJqijABwOeU2Q3N/2qeXV+5U3pmKb9rcHBWqJ4d2Vcsg3wvsDcDZUEYAuKzjhSV6eWW65m3IkmFI/t4eGn9tR/356nby8fQwOx4AO1FGALi8rQfyNP3LbUrKOilJatesgaYN66aBnVuYGwyAXSgjAOoFm83QouSDivt6h44WFEuSBnVtoadu6qY2TRuYnA7A+VBGANQrp4pK9e+EXfrg+70qsxny9rTqgT+010MDOsrPm0s3gDOijACol3YfPqWnv0zVut1HJUmhQb76e2w3DY1sKYvFYnI6AL9GGQFQbxmGoRXbc/TskjQdPHlGktS3fVPNuCVCnYIDTE4H4CzKCIB670xJud5as0dvrdmj4jKbPKwWje7bRhMHdVKQn5fZ8QC3RxkB4Db2Hz+tZ5ekamVqriSpWUNvPXFjF/0xJkxWK5duALNQRgC4nbU7j+jpr7Yr40ihJKlneCPNuDlCUeGNzA0GuCnKCAC3VFJm0+wfMvXaN78swHdnr3A9cWNnNW3oY3Y8wK1QRgC4tcP5RXrh6x36PPmgJCnA11OTru+kP13ZhgX4gDpCGQEASRv3Hte0L7Yr9VcL8D19c4SuZAE+oNZRRgDgZ+U2Q/M2ZOnllek6ebpiAb6beoTo77FdFRLkZ3I6oP6ijADAb5woLNG/VqVr7k9ZshmSn1fFAnz3X8MCfEBtoIwAQDW2HczT019u18Z9JyRJbZv6a9qwbrq2S7DJyYD6hTICAOdhGIYWpxxU3LIdOnyqYgG+a7u00LSbuqltMxbgAy4FyggA2KGguEyvJ+zSe+syKxbg87Dq/mvaafy1HeXv7Wl2PMClUUYAwAG7Dxdoxlfb9d2uigX4QoJ89eTQrrqpRwgL8AE1RBkBAAcZhqGVqbl6dkmqDpyoWIDvyvZN9PTNEerSktcewFGUEQCooaLScr29JkOzEndXLsD3pyvb6NHrWYAPcARlBAAu0oETp/WPpWn6eluOJKlJA289Mbiz7rw8nAX4ADtQRgDgElm366ie/mq7dh8ukCRFhQVpxi3d1ZMF+IDzoowAwCVUWm7Tf3/Yq5nf7FJBcZkk6Y5eYXrixi5qHsACfMC52Pv+zWpRAGAHLw+r7r+mvVY/3l+3x4RJkj7ZdEDXvpyo99dlqrTcZnJCwHXxyQgA1MCmfSc0/ctt2nawYgG+TsEN9fTNEerXoZnJyQDnwWUaAKhl5TZDC/63Xy+t2KETPy/AFxsZoidju6pVIxbgAygjAFBHTp4u0SurdurjH/fJZki+XlaNG9BR/+8P7eXrxQJ8cF+UEQCoY6mH8vX0l9u1Ye9xSVLrJv6adlM3Xde1BXdxhVuq1Qms8fHxatu2rXx9fdWnTx9t2LDhvONnzpypzp07y8/PT+Hh4Xr00UdVVFRUk0MDgNPqFhqoBQ9cqdfu6qngQB9lHT+t+z/cqPtm/08ZRwrMjgc4LYfLyIIFCzRp0iRNnz5dSUlJioqK0uDBg3X48OFzjp87d64mT56s6dOnKy0tTe+9954WLFigJ5988qLDA4CzsVgsuqVnKyU8NkAP9u8gLw+LEtOPaPDMtXrh6x0q/PlrwQB+4fBlmj59+uiKK67QG2+8IUmy2WwKDw/XhAkTNHny5N+NHz9+vNLS0pSQkFD52GOPPaaffvpJ69ats+uYXKYB4KoyjhRoxlepWrPziCSpZaCvpgztopujQrl0g3qvVi7TlJSUaNOmTRo0aNAvv8Bq1aBBg7R+/fpz7tOvXz9t2rSp8lJORkaGli1bpqFDh1Z7nOLiYuXn51fZAMAVtW/eULPvu0Lvjr5crZv4Kye/SI/MT9GId35UWjavbYDkYBk5evSoysvLFRwcXOXx4OBg5eTknHOfkSNH6plnntHVV18tLy8vdejQQQMGDDjvZZq4uDgFBQVVbuHh4Y7EBACnYrFYdH23YK189A967PpO8vWyakPmccX++ztN+2KbTp4uMTsiYKpavwNrYmKinn/+ec2aNUtJSUn6/PPPtXTpUj377LPV7jNlyhTl5eVVbvv376/tmABQ63y9PDThusuU8NgAxUaGyGZIH67fp4EvJ2rehiyV25z+y41ArXBozkhJSYn8/f316aefavjw4ZWPjxkzRidPntQXX3zxu32uueYaXXnllXrppZcqH/v444/1l7/8RQUFBbJaL9yHmDMCoD76YfdRTf9yu3b9vABfZKsgzbglQjGtG5ucDLg0amXOiLe3t3r16lVlMqrNZlNCQoL69u17zn1Onz79u8Lh4VFxEyAXuMUJANSafh2badkj1+ipm7opwMdTWw/m6bZZP+ixhZt1+BS3P4D7cPgyzaRJk/Tuu+/qv//9r9LS0jR27FgVFhbqvvvukySNHj1aU6ZMqRw/bNgwvfnmm5o/f74yMzO1atUqPfXUUxo2bFhlKQEAd+XlYdWfr26n1Y8P0B29Khbg+yzpgK57eY3+810GC/DBLXg6usOIESN05MgRTZs2TTk5OerZs6eWL19eOak1KyuryichU6dOlcVi0dSpU3Xw4EE1b95cw4YN0z/+8Y9L91cAgItrHuCjl+6I0t19WuvpL7dry4E8Pbc0TfP/t18zbo7QVR1ZgA/1F7eDBwAnY7MZWrhxv/65Il3HCyu+aTOke0v9Pbarwhr7m5wOsB9r0wCAi8s7XapXv9mpD9fvrVyA795+7TQ8OlSdgwO4aRqcHmUEAOqJtOx8Tf9yuzZkHq98rH3zBropMkRDe4RQTOC0KCMAUI8YhqEV23P16aYDWrvziEp+NbGVYgJnRRkBgHrqVFGpEtIOa8mW7HMWk9jIEMVSTOAEKCMA4AbsKSZDI0PUpSXFBHWPMgIAbuZsMVm6NVtrdh5RSdmvikmzBortQTFB3aKMAIAbo5jAGVBGAACSKorJ6h0Vl3LOVUyG/jzHhGKCS40yAgD4HXuKydDIEHUNoZjg4lFGAADndbaYLN2SrcTfFJN2zX6Z/EoxQU1RRgAAdqOYoDZQRgAANVJQXKaEtNxqi8nQyJaKjQylmOCCKCMAgItmTzEZGhmibiGBFBP8DmUEAHBJnS0my7Zm69t0igkujDICAKg1vy4mielHVPyrYtK2qX/lfUwoJu6NMgIAqBMXKiZn72NCMXE/lBEAQJ0rKC77+Vs5h6otJkMjQxQRSjFxB5QRAICpzhaTZVuy9W36YYqJG6KMAACcRmFxmRIoJm6HMgIAcErnKyZtzs4xoZjUC5QRAIDTK6ycY0IxqY8oIwAAl3K2mCzbmq3VOygm9QFlBADgsn5dTL5NP6yiUoqJK6KMAADqhfMVk9ZNKorJTT0oJs6IMgIAqHcKi8v0bfovc0zOVUxiI0PUvRXFxBlQRgAA9drZYnJ2jgnFxPlQRgAAbuN0SdXJrxQT50AZAQC4pQsVkyGRLXVTZCjFpA5QRgAAbu90SZm+3XFES7ce+l0xCW/iVzH5lWJSaygjAAD8ytlismxrthJ25J6zmMRGhiiyVRDF5BKhjAAAUA2KSd2w9/3bWpNfHh8fr7Zt28rX11d9+vTRhg0bqh07YMAAWSyW322xsbE1OTQAABfN39tTsT1CFD8qRklPXa/4kTGKjQyRn5eH9h8/o7fXZOjmN77XH176VnHL0rTlwEm5wP+7uyyHPxlZsGCBRo8erbfeekt9+vTRzJkz9cknnyg9PV0tWrT43fjjx4+rpKSk8t/Hjh1TVFSU/vOf/+jee++165h8MgIAqAunS8qUmH5ES7dUTH49U1pe+bOwxn6KjQxRbA8+MbFXrV2m6dOnj6644gq98cYbkiSbzabw8HBNmDBBkydPvuD+M2fO1LRp05Sdna0GDRrYdUzKCACgrlUWk63ZWp127mIyNDJEPcIoJtWplTJSUlIif39/ffrppxo+fHjl42PGjNHJkyf1xRdfXPB3REZGqm/fvnrnnXfsPSxlBABgqjMl5RV3fqWYOMTe929PR37p0aNHVV5eruDg4CqPBwcHa8eOHRfcf8OGDdq2bZvee++9844rLi5WcXFx5b/z8/MdiQkAwCXl5+2hoT8Xjt8WkwMnzujttRl6e22Gwhr/MvmVYmI/h8rIxXrvvfcUGRmp3r17n3dcXFycZsyYUUepAACw32+LSWL6YS35VTF5Z22G3qGYOKTOLtMUFhYqNDRUzzzzjB555JHzHudcn4yEh4dzmQYA4LTOFpOlW7OVcI5LOWcLTJQbFZNauUzj7e2tXr16KSEhobKM2Gw2JSQkaPz48efd95NPPlFxcbHuueeeCx7Hx8dHPj4+jkQDAMBUft4eGhIZoiG/+sTkbDH59ScmrRr5KbaH+xWT86nRV3vHjBmjt99+W71799bMmTO1cOFC7dixQ8HBwRo9erRatWqluLi4Kvtdc801atWqlebPn+9wSCawAgBc1a+Lyeodh3W65JdPTFo18tPQyJaK7RFaL4tJrXwyIkkjRozQkSNHNG3aNOXk5Khnz55avnx55aTWrKwsWa1V76WWnp6udevWaeXKlY4eDgAAl/bbT0zW7DysJT/fx+TgyTN697tMvftdZr0vJufD7eABADDB2WKydGuOEtJyz/mJydDIEPUMb+SyxYS1aQAAcBFFpWcv5dSvYkIZAQDABdWnYkIZAQDAxVUUkyM/fyvn98VkSPeWiu3hvMWEMgIAQD1ytpgs25qtb6opJkN7hCjaiYoJZQQAgHrqfMUkNMi34gZrTlBMKCMAALiBXxeThLRcFTpRMaGMAADgZopKy7Vm5xEt3XLuYjIkMkSxdVhMKCMAALgxe4rJ0MiKYmK11k4xoYwAAABJvxSTZVuz9U1q1WIS8vOlnDsvD1fnlgGX9Li1djt4AADgWny9PDQ4oqUGR7T8XTHJzivSe+sy1bFFw0teRuxFGQEAwI38tpis3VlxH5PBES1Ny0QZAQDATfl6eeiGiJa6wcQiIknWCw8BAACoPZQRAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAEzlEqv2GoYhScrPzzc5CQAAsNfZ9+2z7+PVcYkycurUKUlSeHi4yUkAAICjTp06paCgoGp/bjEuVFecgM1m06FDhxQQECCLxXLJfm9+fr7Cw8O1f/9+BQYGXrLfW19xvuzHubIf58p+nCv7ca7sV5vnyjAMnTp1SqGhobJaq58Z4hKfjFitVoWFhdXa7w8MDOTJ6gDOl/04V/bjXNmPc2U/zpX9autcne8TkbOYwAoAAExFGQEAAKZy6zLi4+Oj6dOny8fHx+woLoHzZT/Olf04V/bjXNmPc2U/ZzhXLjGBFQAA1F9u/ckIAAAwH2UEAACYijICAABMRRkBAACmqvdlJD4+Xm3btpWvr6/69OmjDRs2nHf8J598oi5dusjX11eRkZFatmxZHSU1nyPnavbs2bJYLFU2X1/fOkxrnrVr12rYsGEKDQ2VxWLR4sWLL7hPYmKiYmJi5OPjo44dO2r27Nm1ntMZOHquEhMTf/e8slgsysnJqZvAJoqLi9MVV1yhgIAAtWjRQsOHD1d6evoF93PH16yanCt3fc1688031aNHj8obmvXt21dff/31efcx4zlVr8vIggULNGnSJE2fPl1JSUmKiorS4MGDdfjw4XOO/+GHH3T33Xfrz3/+s5KTkzV8+HANHz5c27Ztq+Pkdc/RcyVV3K0vOzu7ctu3b18dJjZPYWGhoqKiFB8fb9f4zMxMxcbGauDAgUpJSdHEiRN1//33a8WKFbWc1HyOnquz0tPTqzy3WrRoUUsJnceaNWs0btw4/fjjj1q1apVKS0t1ww03qLCwsNp93PU1qybnSnLP16ywsDC98MIL2rRpkzZu3Khrr71Wt9xyi7Zv337O8aY9p4x6rHfv3sa4ceMq/11eXm6EhoYacXFx5xx/5513GrGxsVUe69Onj/HAAw/Uak5n4Oi5+uCDD4ygoKA6Sue8JBmLFi0675gnnnjCiIiIqPLYiBEjjMGDB9diMudjz7n69ttvDUnGiRMn6iSTMzt8+LAhyVizZk21Y9z5NevX7DlXvGb9onHjxsZ//vOfc/7MrOdUvf1kpKSkRJs2bdKgQYMqH7NarRo0aJDWr19/zn3Wr19fZbwkDR48uNrx9UVNzpUkFRQUqE2bNgoPDz9v03Z37vq8uhg9e/ZUSEiIrr/+en3//fdmxzFFXl6eJKlJkybVjuG5VcGecyXxmlVeXq758+ersLBQffv2PecYs55T9baMHD16VOXl5QoODq7yeHBwcLXXn3NychwaX1/U5Fx17txZ77//vr744gt9/PHHstls6tevnw4cOFAXkV1Kdc+r/Px8nTlzxqRUzikkJERvvfWWPvvsM3322WcKDw/XgAEDlJSUZHa0OmWz2TRx4kRdddVV6t69e7Xj3PU169fsPVfu/Jq1detWNWzYUD4+PnrwwQe1aNEidevW7ZxjzXpOucSqvXA+ffv2rdKs+/Xrp65du+rtt9/Ws88+a2IyuLLOnTurc+fOlf/u16+f9uzZo1dffVUfffSRicnq1rhx47Rt2zatW7fO7ChOz95z5c6vWZ07d1ZKSory8vL06aefasyYMVqzZk21hcQM9faTkWbNmsnDw0O5ublVHs/NzVXLli3PuU/Lli0dGl9f1ORc/ZaXl5eio6O1e/fu2ojo0qp7XgUGBsrPz8+kVK6jd+/ebvW8Gj9+vJYsWaJvv/1WYWFh5x3rrq9ZZzlyrn7LnV6zvL291bFjR/Xq1UtxcXGKiorSa6+9ds6xZj2n6m0Z8fb2Vq9evZSQkFD5mM1mU0JCQrXXyvr27VtlvCStWrWq2vH1RU3O1W+Vl5dr69atCgkJqa2YLstdn1eXSkpKils8rwzD0Pjx47Vo0SKtXr1a7dq1u+A+7vrcqsm5+i13fs2y2WwqLi4+589Me07V6vRYk82fP9/w8fExZs+ebaSmphp/+ctfjEaNGhk5OTmGYRjGn/70J2Py5MmV47///nvD09PTePnll420tDRj+vTphpeXl7F161az/oQ64+i5mjFjhrFixQpjz549xqZNm4y77rrL8PX1NbZv327Wn1BnTp06ZSQnJxvJycmGJOOVV14xkpOTjX379hmGYRiTJ082/vSnP1WOz8jIMPz9/Y2//vWvRlpamhEfH294eHgYy5cvN+tPqDOOnqtXX33VWLx4sbFr1y5j69atxiOPPGJYrVbjm2++MetPqDNjx441goKCjMTERCM7O7tyO336dOUYXrMq1ORcuetr1uTJk401a9YYmZmZxpYtW4zJkycbFovFWLlypWEYzvOcqtdlxDAM4/XXXzdat25teHt7G7179zZ+/PHHyp/179/fGDNmTJXxCxcuNDp16mR4e3sbERERxtKlS+s4sXkcOVcTJ06sHBscHGwMHTrUSEpKMiF13Tv79dPfbmfPz5gxY4z+/fv/bp+ePXsa3t7eRvv27Y0PPvigznObwdFz9eKLLxodOnQwfH19jSZNmhgDBgwwVq9ebU74Onau8ySpynOF16wKNTlX7vqa9X//939GmzZtDG9vb6N58+bGddddV1lEDMN5nlMWwzCM2v3sBQAAoHr1ds4IAABwDZQRAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJjq/wMKY89yiQ9JMgAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "class CustomCallback(keras.callbacks.Callback):\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        keys = list(logs.keys())\n",
        "        if(epoch % EVAL_STEPS == EVAL_STEPS-1):\n",
        "          # Evaluate\n",
        "          display(HTML(inference_test(test_image)))\n",
        "          make_predictions()\n",
        "\n",
        "#history = pali_gemma_lm.fit(train_data, epochs=TRAIN_STEPS, callbacks=[CustomCallback()])\n",
        "history = pali_gemma_lm.fit(train_data, epochs=TRAIN_STEPS)\n",
        "plt.plot(history.history['loss'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dbfa8168",
      "metadata": {
        "id": "37fa359da1d9"
      },
      "source": [
        "## Output\n",
        "\n",
        "The validation data for this notebook consists of just 10 images. In normal code, you would likely have many more data points for validation, but for this notebook, run the following code to generate descriptions for all 10 images. After tuning the model, these descriptions should be very similar in form and content coverage to the descriptions included with the training data that you looked at earlier in this notebook.\n",
        "\n",
        "Run the below code to generate descriptions for the validation data set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "35f5080d",
      "metadata": {
        "id": "321ca68e4b44"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A brown cow stands proudly on a beach, its ears pricks. The cow has a white spot on its face. The cow has a white spot on its face. The grass is brown and the sky is clear. The cow is standing on the beach, its ears pricks. The</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Inference Result\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A person wearing a red blazer and black pants, a red blazer, and a white shirt. The blazer is long and the pants are long. The blazer has a long red button. The blazer has a long red button. The pants are black. The blazer has a long red button.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A man stands in a park, wearing a denim jacket and white shoes. The man wears a white shirt, brown pants, and white shoes. The jacket is blue. The pants are brown. The shoes are white. The jacket is blue. The shirt is white. The man is standing</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A woman wearing a white dress with a red flower pattern, a black skirt, and a brown belt. The dress is long and has a white skirt. The woman is wearing a brown belt, a brown leather purse, a brown paper bag, a brown basket, and a brown paper basket</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A person holds a pink sweater with a red heart on it. The sweater has a white collar, long sleeves, and long sleeves. The sweater has a red heart on it. The sweater has a black heart on it. The sweater has a red heart on it. The sweater has a</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display(HTML(inference_test(test_image)))\n",
        "make_predictions()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "[PaliGemma_2]Finetune_with_Keras.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
